249 research outputs found
Relaxed Sparse Eigenvalue Conditions for Sparse Estimation via Non-convex Regularized Regression
Non-convex regularizers usually improve the performance of sparse estimation
in practice. To prove this fact, we study the conditions of sparse estimations
for the sharp concave regularizers which are a general family of non-convex
regularizers including many existing regularizers. For the global solutions of
the regularized regression, our sparse eigenvalue based conditions are weaker
than that of L1-regularization for parameter estimation and sparseness
estimation. For the approximate global and approximate stationary (AGAS)
solutions, almost the same conditions are also enough. We show that the desired
AGAS solutions can be obtained by coordinate descent (CD) based methods.
Finally, we perform some experiments to show the performance of CD methods on
giving AGAS solutions and the degree of weakness of the estimation conditions
required by the sharp concave regularizers
Multi-Stage Multi-Task Feature Learning
Multi-task sparse feature learning aims to improve the generalization
performance by exploiting the shared features among tasks. It has been
successfully applied to many applications including computer vision and
biomedical informatics. Most of the existing multi-task sparse feature learning
algorithms are formulated as a convex sparse regularization problem, which is
usually suboptimal, due to its looseness for approximating an -type
regularizer. In this paper, we propose a non-convex formulation for multi-task
sparse feature learning based on a novel non-convex regularizer. To solve the
non-convex optimization problem, we propose a Multi-Stage Multi-Task Feature
Learning (MSMTFL) algorithm; we also provide intuitive interpretations,
detailed convergence and reproducibility analysis for the proposed algorithm.
Moreover, we present a detailed theoretical analysis showing that MSMTFL
achieves a better parameter estimation error bound than the convex formulation.
Empirical studies on both synthetic and real-world data sets demonstrate the
effectiveness of MSMTFL in comparison with the state of the art multi-task
sparse feature learning algorithms.Comment: The short version appears in NIPS 201
Deep Defense: Training DNNs with Improved Adversarial Robustness
Despite the efficacy on a variety of computer vision tasks, deep neural
networks (DNNs) are vulnerable to adversarial attacks, limiting their
applications in security-critical systems. Recent works have shown the
possibility of generating imperceptibly perturbed image inputs (a.k.a.,
adversarial examples) to fool well-trained DNN classifiers into making
arbitrary predictions. To address this problem, we propose a training recipe
named "deep defense". Our core idea is to integrate an adversarial
perturbation-based regularizer into the classification objective, such that the
obtained models learn to resist potential attacks, directly and precisely. The
whole optimization problem is solved just like training a recursive network.
Experimental results demonstrate that our method outperforms training with
adversarial/Parseval regularizations by large margins on various datasets
(including MNIST, CIFAR-10 and ImageNet) and different DNN architectures. Code
and models for reproducing our results are available at
https://github.com/ZiangYan/deepdefense.pytorchComment: Accepted by NeurIPS 201
Sparse DNNs with Improved Adversarial Robustness
Deep neural networks (DNNs) are computationally/memory-intensive and
vulnerable to adversarial attacks, making them prohibitive in some real-world
applications. By converting dense models into sparse ones, pruning appears to
be a promising solution to reducing the computation/memory cost. This paper
studies classification models, especially DNN-based ones, to demonstrate that
there exists intrinsic relationships between their sparsity and adversarial
robustness. Our analyses reveal, both theoretically and empirically, that
nonlinear DNN-based classifiers behave differently under attacks from
some linear ones. We further demonstrate that an appropriately higher model
sparsity implies better robustness of nonlinear DNNs, whereas over-sparsified
models can be more difficult to resist adversarial examples.Comment: l1 regularization on weights --> l1 regularization on activation
Aligning where to see and what to tell: image caption with region-based attention and scene factorization
Recent progress on automatic generation of image captions has shown that it
is possible to describe the most salient information conveyed by images with
accurate and meaningful sentences. In this paper, we propose an image caption
system that exploits the parallel structures between images and sentences. In
our model, the process of generating the next word, given the previously
generated ones, is aligned with the visual perception experience where the
attention shifting among the visual regions imposes a thread of visual
ordering. This alignment characterizes the flow of "abstract meaning", encoding
what is semantically shared by both the visual scene and the text description.
Our system also makes another novel modeling contribution by introducing
scene-specific contexts that capture higher-level semantic information encoded
in an image. The contexts adapt language models for word generation to specific
scene types. We benchmark our system and contrast to published results on
several popular datasets. We show that using either region-based attention or
scene-specific contexts improves systems without those components. Furthermore,
combining these two modeling ingredients attains the state-of-the-art
performance
Recent Advances in Large Margin Learning
This paper serves as a survey of recent advances in large margin training and
its theoretical foundations, mostly for (nonlinear) deep neural networks (DNNs)
that are probably the most prominent machine learning models for large-scale
data in the community over the past decade. We generalize the formulation of
classification margins from classical research to latest DNNs, summarize
theoretical connections between the margin, network generalization, and
robustness, and introduce recent efforts in enlarging the margins for DNNs
comprehensively. Since the viewpoint of different methods is discrepant, we
categorize them into groups for ease of comparison and discussion in the paper.
Hopefully, our discussions and overview inspire new research work in the
community that aim to improve the performance of DNNs, and we also point to
directions where the large margin principle can be verified to provide
theoretical evidence why certain regularizations for DNNs function well in
practice. We managed to shorten the paper such that the crucial spirit of large
margin learning and related methods are better emphasized.Comment: 8 pages, 3 figure
Weakly- and Semi-Supervised Object Detection with Expectation-Maximization Algorithm
Object detection when provided image-level labels instead of instance-level
labels (i.e., bounding boxes) during training is an important problem in
computer vision, since large scale image datasets with instance-level labels
are extremely costly to obtain. In this paper, we address this challenging
problem by developing an Expectation-Maximization (EM) based object detection
method using deep convolutional neural networks (CNNs). Our method is
applicable to both the weakly-supervised and semi-supervised settings.
Extensive experiments on PASCAL VOC 2007 benchmark show that (1) in the weakly
supervised setting, our method provides significant detection performance
improvement over current state-of-the-art methods, (2) having access to a small
number of strongly (instance-level) annotated images, our method can almost
match the performace of the fully supervised Fast RCNN. We share our source
code at https://github.com/ZiangYan/EM-WSD.Comment: 9 page
A General Iterative Shrinkage and Thresholding Algorithm for Non-convex Regularized Optimization Problems
Non-convex sparsity-inducing penalties have recently received considerable
attentions in sparse learning. Recent theoretical investigations have
demonstrated their superiority over the convex counterparts in several sparse
learning settings. However, solving the non-convex optimization problems
associated with non-convex penalties remains a big challenge. A commonly used
approach is the Multi-Stage (MS) convex relaxation (or DC programming), which
relaxes the original non-convex problem to a sequence of convex problems. This
approach is usually not very practical for large-scale problems because its
computational cost is a multiple of solving a single convex problem. In this
paper, we propose a General Iterative Shrinkage and Thresholding (GIST)
algorithm to solve the nonconvex optimization problem for a large class of
non-convex penalties. The GIST algorithm iteratively solves a proximal operator
problem, which in turn has a closed-form solution for many commonly used
penalties. At each outer iteration of the algorithm, we use a line search
initialized by the Barzilai-Borwein (BB) rule that allows finding an
appropriate step size quickly. The paper also presents a detailed convergence
analysis of the GIST algorithm. The efficiency of the proposed algorithm is
demonstrated by extensive experiments on large-scale data sets
Instance-level Semisupervised Multiple Instance Learning
Multiple instance learning (MIL) is a branch of machine learning that attempts to learn information from bags of instances. Many real-world applications such as localized content-based image retrieval and text categorization can be viewed as MIL problems. In this paper, we propose a new graph-based semi-supervised learning approach for multiple instance learning. By defining an instance-level graph on the data, we first propose a new approach to construct an optimization framework for multiple instance semi-supervised learning, and derive an efficient way to overcome the non-convexity of MIL. We empirically show that our method outperforms state-of-the-art MIL algorithms on several real-world data sets
An In-field Automatic Wheat Disease Diagnosis System
Crop diseases are responsible for the major production reduction and economic
losses in agricultural industry world- wide. Monitoring for health status of
crops is critical to control the spread of diseases and implement effective
management. This paper presents an in-field automatic wheat disease diagnosis
system based on a weakly super- vised deep learning framework, i.e. deep
multiple instance learning, which achieves an integration of identification for
wheat diseases and localization for disease areas with only image-level
annotation for training images in wild conditions. Furthermore, a new in-field
image dataset for wheat disease, Wheat Disease Database 2017 (WDD2017), is
collected to verify the effectiveness of our system. Under two different
architectures, i.e. VGG-FCN-VD16 and VGG-FCN-S, our system achieves the mean
recognition accuracies of 97.95% and 95.12% respectively over 5-fold
cross-validation on WDD2017, exceeding the results of 93.27% and 73.00% by two
conventional CNN frameworks, i.e. VGG-CNN-VD16 and VGG-CNN-S. Experimental
results demonstrate that the proposed system outperforms conventional CNN
architectures on recognition accuracy under the same amount of parameters,
meanwhile main- taining accurate localization for corresponding disease areas.
Moreover, the proposed system has been packed into a real-time mobile app to
provide support for agricultural disease diagnosis.Comment: 15 page
- …